build: add option to create JSON info files #2192

aparcar · 2019-07-03T13:04:26Z

The JSON info files contain details about the created firmware images
per device and are stored next to the created images.

The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.

An example of openwrt-ramips-rt305x-aztech_hw550-3g.json

{
  "id": "aztech_hw550-3g",
  "image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g",
  "images": [
    {
      "name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin",
      "sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20",
      "type": "sysupgrade"
    }
  ],
  "metadata_version": 1,
  "supported_devices": [
    "aztech,hw550-3g",
    "hw550-3g"
  ],
  "target": "ramips/rt305x",
  "titles": [
    {
      "model": "HW550-3G",
      "vendor": "Aztech"
    },
    {
      "model": "ALL0239-3G",
      "vendor": "Allnet"
    }
  ],
  "version_commit": "r10920+123-0cc87b3bac",
  "version_number": "SNAPSHOT"
}

Signed-off-by: Paul Spooren mail@aparcar.org

adschm · 2019-07-04T18:56:55Z

After each build process all JSON files are merged in a single devices.json file only containing title and device ID.

Had to think twice about that: JSON files are NOT merged, but selective information about them is collected in devices.json ...

adschm · 2019-07-04T18:59:10Z

"8devices Carambola2": "8dev_carambola2"

Do you have a reason why you use the friendly name as key and the ID as value? I would have done it the other way around, so the ID is used as ID (although it technically won't matter if I got your use-case right)

scripts/add_json.sh

aparcar · 2019-07-05T09:55:59Z

After each build process all JSON files are merged in a single devices.json file only containing title and device ID.

Had to think twice about that: JSON files are NOT merged, but selective information about them is collected in devices.json ...

devices.json is like a map, on what devices are available.

Do you have a reason why you use the friendly name as key and the ID as value? I would have done it the other way around, so the ID is used as ID (although it technically won't matter if I got your use-case right)

Idea is to search trough the keys via a fuzzy search, once a device is found more detailed information are gathered via the <device_id>.json

adschm · 2019-07-05T11:08:02Z

devices.json is like a map, on what devices are available.

I'm aware of that, I just wanted to advice on changing the part in the commit message/PR description where you say "merged" to make it clearer.

adschm · 2019-07-05T11:09:20Z

Idea is to search trough the keys via a fuzzy search, once a device is found more detailed information are gathered via the <device_id>.json

Well, I still would exchange key and value and then search through the values.

aparcar · 2019-07-05T15:12:57Z

exchanged key and value, also clarified the commit message

aparcar · 2019-07-07T12:21:45Z

@jow- please have a look

aparcar · 2019-07-07T12:59:15Z

Here is an example for resulting files https://chef.libremesh.org/download/json/

aparcar · 2019-07-13T20:42:10Z

@blogic I moved the output to $(TOPDIR)/bin/json/. Would that work for you?

lynxis · 2019-07-15T11:39:50Z

I would like to have a hash (sha256) of glinet_gl-ar150.json into the devices.json.
If we have this, we could sign devices.json and have a valid path down to the image. I see value in this, even if the images are signed itself.

I'm also missing a version field of the json structure. I mean the api/specification of the json field.

aparcar · 2019-07-15T13:42:05Z

The approach was changed, now maps.json and overview.json are ingame to solve two issues at once: vanilla firmware upgrades and firmware search! The script became to complex to use purse shell, but as Python3 becomes (is?) a buildroot dependency anyway, that shouldn't cause any problem.

If a (to be designed) release.json file is signed and contains checksums of the maps.json file, a secure chain is created as long as sha256 isn't broken.

aparcar · 2019-07-16T16:05:42Z

Now uses two Python3 scripts instead of messy shell scripts. Also a map file is created per subtarget.

aparcar · 2019-07-16T16:34:44Z

Now also handles ALT titles

Utilises the JSON schema generated after build as suggested in: openwrt/openwrt#2192 (comment) Ability to apply fresh images (default packages only, from openwrt) or to apply image with installed packages (from attendedsysupgrade server) Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>

aparcar · 2019-08-27T21:10:38Z

Please see the outcome of this years GSoC here. It's a firmware wizard which utilizes the JSON produced in this PR to allow users simple selection of firmware. It also supports images on demand for custom package selections, please have a look.

While unmerged, patch it here openwrt/openwrt#2192 Signed-off-by: Paul Spooren <mail@aparcar.org>

This is a proof of conecpt and requires further work, it show how a device can find upgrades only in a secure way, without any extra packages, especially nothing HTTPS related. The online.sh "library" can be used for CLIs or ubus calls for Luci or other webinterfaces to come. Details about the JSON structure are explained in another PR[0]. To try the script, run `sysupgrade-online --search` or `--unattended`, the former searches for a sysupgrade (only snapshots currently), the latter searches and passes results to a download function, which then downloaded the sysupgrade. Ideally the sysupgrade would be verified aditionally via usign. [0]: openwrt#2192 Signed-off-by: Paul Spooren <mail@aparcar.org>

aparcar · 2019-09-02T19:13:46Z

I created a proof of concept on how devices could find securely firmware upgrades.

scripts/json_add_image_info.py

jow- · 2019-09-13T09:19:33Z

Where do these file end up? Are the JSON files really put next to the image files?
Does the merging really need to be part of buildroot? Can't this be done elsewhere?

aparcar · 2019-09-14T00:44:19Z

It's not important to store the files next to the images, but putting them next to buildinfo and manifest files seemed reasonable to me. If you'd prefer so, I'd move the files to a different folder. However I'd really be in favor to have this files created by the buildbot. I see three use cases which all require up to date JSON files.

A firmware wizard to allow easy image retrieval for beginners.
A way to search, download and install sysupgrades from command line or web interface without a complex backend
A device overview page like the wiki containing up to date download link without manual updates

Removing the merging process from the buildroot be possible via the json overview but still, I'd much more like to see these files officially distributed at downloads.openwrt.org.

aparcar · 2019-09-19T22:44:01Z

@jow- I removed the merging for now as this can be done somewhere else as well. Also enabled it only for buildbots by default.

While unmerged, patch it here openwrt/openwrt#2192 Signed-off-by: Paul Spooren <mail@aparcar.org>

The JSON info files contain details about the created firmware images per device and are stored next to the created images. The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some device/image meta data as well as a list of created firmware images. An example of openwrt-ramips-rt305x-aztech_hw550-3g.json { "id": "aztech_hw550-3g", "image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g", "images": [ { "name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin", "sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20", "type": "sysupgrade" } ], "metadata_version": 1, "supported_devices": [ "aztech,hw550-3g", "hw550-3g" ], "target": "ramips/rt305x", "titles": [ { "model": "HW550-3G", "vendor": "Aztech" }, { "model": "ALL0239-3G", "vendor": "Allnet" } ], "version_commit": "r10920+123-0cc87b3bac", "version_number": "SNAPSHOT" } Signed-off-by: Paul Spooren <mail@aparcar.org>

dangowrt · 2019-09-29T12:34:41Z

Cherry-picked on master. Thanks!

jow- · 2019-10-11T10:37:13Z

config/Config-build.in

@@ -7,6 +7,13 @@

 menu "Global build settings"

+	config JSON_ADD_IMAGE_INFO
+		bool "Create JSON info files per build image"


Should that be per built image ?

Yes, actually even files per build profile, as a profile may contain multiple images (factory, sysupgrade,...). Will update that

jow- · 2019-10-11T10:42:36Z

config/Config-build.in

+		bool "Create JSON info files per build image"
+		default BUILDBOT
+		help
+		  The JSON info files contain information about the device and


Suggestion:

The generated JSON files, named after the base name of image (e.g. "openwrt-ath79-generic-8dev_carambola2..."), will be stored in the same directory as built image files.

Each JSON file contains meta information about the devices supported by a family of images
and the image files belonging to a particular family.

Thank you I will update that

aparcar · 2019-10-12T19:59:03Z

@jow- The currently missing json files on download.openwrt.org are due to the prepare step happening after target/install. I wrote that earlier in IRC

would it be possible to run "make prepare" before "target/install"? from my understanding it should just generate buildinfo files. However it seem to trigger "Device/Build" from include/image.mk which clean up the json files

I don't see a reason why prepare shouldn't run before target/install, it is only required to run after the feeds update to have correct commits per feed in feeds.buildinfo.

aparcar force-pushed the CREATE_JSON branch 4 times, most recently from 2a2bda1 to 40ed190 Compare July 4, 2019 15:05

aparcar changed the title ~~[wip]build: add option to create JSON info files~~ build: add option to create JSON info files Jul 4, 2019

adschm reviewed Jul 4, 2019

View reviewed changes

scripts/add_json.sh Outdated Show resolved Hide resolved

dedeckeh added the build/scripts/tools pull request/issues for build, scripts and tools related changes label Jul 4, 2019

aparcar force-pushed the CREATE_JSON branch 2 times, most recently from 86a128f to 4fa6e5f Compare July 5, 2019 10:07

aparcar force-pushed the CREATE_JSON branch from 4fa6e5f to f4724c9 Compare July 5, 2019 15:12

aparcar force-pushed the CREATE_JSON branch from f4724c9 to bd2277d Compare July 7, 2019 12:49

aparcar force-pushed the CREATE_JSON branch from bd2277d to e69c006 Compare July 13, 2019 20:38

aparcar force-pushed the CREATE_JSON branch from e69c006 to 9edbe69 Compare July 13, 2019 20:45

aparcar force-pushed the CREATE_JSON branch from 9edbe69 to 7c3023b Compare July 15, 2019 13:37

aparcar force-pushed the CREATE_JSON branch 2 times, most recently from f8b7eff to 4d58d3b Compare July 16, 2019 16:04

aparcar force-pushed the CREATE_JSON branch from 4d58d3b to 30e0216 Compare July 16, 2019 16:32

sudhanshug16 mentioned this pull request Aug 26, 2019

[WIP] Sysupgrade: LuCI application for system updates openwrt/luci#3009

Closed

aparcar added a commit to aparcar/openwrt-metabuilder that referenced this pull request Sep 1, 2019

Backport JSON info files patch

eeb3086

While unmerged, patch it here openwrt/openwrt#2192 Signed-off-by: Paul Spooren <mail@aparcar.org>

aparcar mentioned this pull request Sep 1, 2019

[RFC][POC]base-files: add sysupgrade-online script as POC #2366

Closed

aparcar force-pushed the CREATE_JSON branch from 47776f6 to 65f409d Compare September 2, 2019 19:06

dangowrt reviewed Sep 12, 2019

View reviewed changes

scripts/json_add_image_info.py Outdated Show resolved Hide resolved

aparcar force-pushed the CREATE_JSON branch 2 times, most recently from f013c3e to 94610a3 Compare September 19, 2019 22:42

aparcar force-pushed the CREATE_JSON branch from 94610a3 to c3767e8 Compare September 26, 2019 09:24

aparcar added a commit to aparcar/openwrt-metabuilder that referenced this pull request Sep 28, 2019

Backport JSON info files patch

861982f

While unmerged, patch it here openwrt/openwrt#2192 Signed-off-by: Paul Spooren <mail@aparcar.org>

aparcar force-pushed the CREATE_JSON branch from c3767e8 to 3934da4 Compare September 28, 2019 21:44

dangowrt approved these changes Sep 29, 2019

View reviewed changes

dangowrt closed this Sep 29, 2019

jow- reviewed Oct 11, 2019

View reviewed changes

aparcar mentioned this pull request Jun 30, 2020

[19.07]build: create JSON files containing image info #3055

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: add option to create JSON info files #2192

build: add option to create JSON info files #2192

aparcar commented Jul 3, 2019 •

edited

adschm commented Jul 4, 2019

adschm commented Jul 4, 2019

aparcar commented Jul 5, 2019

adschm commented Jul 5, 2019 •

edited

adschm commented Jul 5, 2019

aparcar commented Jul 5, 2019

aparcar commented Jul 7, 2019

aparcar commented Jul 7, 2019

aparcar commented Jul 13, 2019

lynxis commented Jul 15, 2019 •

edited

aparcar commented Jul 15, 2019

aparcar commented Jul 16, 2019

aparcar commented Jul 16, 2019

aparcar commented Aug 27, 2019

aparcar commented Sep 2, 2019

jow- commented Sep 13, 2019

aparcar commented Sep 14, 2019

aparcar commented Sep 19, 2019

dangowrt commented Sep 29, 2019

jow- Oct 11, 2019

aparcar Oct 12, 2019

jow- Oct 11, 2019

aparcar Oct 12, 2019

aparcar commented Oct 12, 2019

build: add option to create JSON info files #2192

build: add option to create JSON info files #2192

Conversation

aparcar commented Jul 3, 2019 • edited

adschm commented Jul 4, 2019

adschm commented Jul 4, 2019

aparcar commented Jul 5, 2019

adschm commented Jul 5, 2019 • edited

adschm commented Jul 5, 2019

aparcar commented Jul 5, 2019

aparcar commented Jul 7, 2019

aparcar commented Jul 7, 2019

aparcar commented Jul 13, 2019

lynxis commented Jul 15, 2019 • edited

aparcar commented Jul 15, 2019

aparcar commented Jul 16, 2019

aparcar commented Jul 16, 2019

aparcar commented Aug 27, 2019

aparcar commented Sep 2, 2019

jow- commented Sep 13, 2019

aparcar commented Sep 14, 2019

aparcar commented Sep 19, 2019

dangowrt commented Sep 29, 2019

jow- Oct 11, 2019

Choose a reason for hiding this comment

aparcar Oct 12, 2019

Choose a reason for hiding this comment

jow- Oct 11, 2019

Choose a reason for hiding this comment

aparcar Oct 12, 2019

Choose a reason for hiding this comment

aparcar commented Oct 12, 2019

aparcar commented Jul 3, 2019 •

edited

adschm commented Jul 5, 2019 •

edited

lynxis commented Jul 15, 2019 •

edited